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ABSTRACT 

Any large organization whose mission is to design and develop systems for humans, evaluate systems for humans, and train humans needs a well-developed integration and process plan to 
deal with the challenges that arise from managing multiple subsystems. Human capabilities, skills, and needs must be considered early in the design and development process, and must be 
continuously considered throughout the development lifecycle. This integration of human needs within system design is typically formalized through a Human-Systems Integration (HSI) 
program. By having an HSI program, an institution or organization can reduce lifecycle costs and increase the efficiency, usability, and quality of its products because human needs have been 
considered from the beginning. 


BENCHMARKING HSI PRACTICES 


INTRODUCTION 

Human-Systems Integration (HSI) emphasizes human considerations as the top 
priority in systems design to optimize the fully integrated system’s (i.e., human and 
machine’s) performance and often to reduce lifecycle costs. HSI at NASA is a 
multidisciplinary field of study composed of several user-related areas, including: 
Human Factors Engineering (HFE), System Safety, Health Hazards, Manpower & 
Personnel, Training, and Habitability. 



RESEARCH METHODS 

One purpose of the project is to benchmark DoD and other organizations’ HSI 
practices. This is being accomplished by organizing meetings with DoD 
representatives who use MANPRINT (Manpower and Personnel Integration), 
SEAPRINT (Systems Engineering, Acquisition and Personnel Integration), and 
AIRPRINT (Airman Performance Integration), meeting with other organizations (e.g., 
industry), studying their processes, reviewing their documents, and meeting with 
contractors who implement these HSI requirements. The purpose of these activities is 
to learn how the requirements are implemented in the final products. Agency and 
Johnson Space Center (JSC) programmatic and Systems Engineering documented 
processes and practices will also be reviewed, and a gap analysis will be developed to 
identify where HSI should be a part of the process. 

INTERVIEWS AND DISCUSSIONS 


Interviews and discussions will be conducted with personnel involved in HSI practices 
in other organizations to learn about their processes. An interview questionnaire has 
been put together covering the different topics. 



FUTURE RESEARCH DIRECTIONS 

Directed research must specifically investigate lifecycle costing and identify the 
contributions that HSI can make in this arena for the Agency. For example, in the DoD, 
HSI was advocated on the basis of cost savings, not on the promise alone of producing 
superior user-interfaces. It is essential that DoD and NASA work together to develop 
metrics that can be used during development that provide indications of operational 
efficiencies and inefficiencies, preferably using lifecycle cost as one of the desired 
output metrics. Our research should pursue identification of a forward plan for 
integration of lifecycle costing into the Agency for the sake of demonstrating HSI’s 
usefulness in this area. 

The Human Factors Engineering processes have to be inserted into the NASA project 
lifecycle to ensure a Human Centered Design process. Below is a flowchart that 
illustrates possible insertion points: 



HUMAN PERFORMANCE METRICS 


Human-in-the-loop evaluation 

A human-in-the-loop evaluation is any evaluation that includes a human, whether in an active or 
passive capacity in the participant role. The active human-in -the -loop means that the human’s 
actions are being evaluated in some capacity. The passive human-in-the-loop means that the 
human is providing passive data (waste, or other physiological outputs). The human as 
participant means that the human is providing the data in which case human performance can be 
captured. Using standardized metrics helps to use data collected across evaluations. 

From an HSI point of view, for humans to be truly included as a “system”, designers and 
evaluators must be able to quantify their performance. To attain this, human performance 
metrics have to be defined and also standardized within an organization so that data can be 
shared across systems. 

Human performance metrics 

Human factors metrics depend on context, phase, and cost. Performance measures can be used 
in formative or summative evaluations, and these are determined by the research goal. Human 
factors performance measures can be usage or predictive metrics and these define the method 
used. 
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Human performance measures database 

A database is being developed that will contain all the metrics with recommendations for users. 
The purpose of the Human Performance Metrics database is to address the need for standard 
metrics to be collected whenever humans are included in testing. It is always advantageous to 
collect human performance measures no matter how small the number of participants per test. 
This data can become part of an archive of data which may be later combined with other 
archives to be correlated, and compared. 



